Inputs to virtual reality devices from touch surface devices

ABSTRACT

In example implementations, a virtual reality (VR) system is provided. The VR system includes a head mounted display (HMD), a hand wearable interface, a wireless communication interface, and a processor. The HMD is to display a computer generated artificial world. The wireless communication interface is to establish a wireless communication path. The processor is communicatively coupled to the HMD, the hand wearable interface, and the wireless communication interface. The processor is to receive an indication that a touch screen device is located from a locator device in the HMD or the hand wearable device, to establish a wireless connection to the touch screen device via the wireless communication interface, and to receive an input via the touch screen device.

BACKGROUND

Virtual reality (VR) systems are wearable interactive systems that allowa user to experience an artificial world. The user may visually see acomputer generated world through a display of the VR system. The VRsystem may provide entertainment, simulations, and the like. Forexample, the artificial world may be part of a video game forentertainment. In another example, the VR world may be a simulation totrain an employee for a procedure or process in a corporate setting.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example of a virtual reality system witha touch surface device of the present disclosure;

FIG. 2 illustrates a block diagram of the virtual reality system thepresent disclosure;

FIG. 3 illustrates an example display of the touch surface device in avirtual reality display of the virtual reality system of the presentdisclosure;

FIG. 4 is a flow chart of an example method for connecting a virtualreality device to a touch surface device to receive an input from thetouch surface device; and

FIG. 5 is a block diagram of an example non-transitory computer readablestorage medium storing instructions executed by a processor of thepresent disclosure.

DETAILED DESCRIPTION

Examples described herein provide a virtual reality (VR) system that canconnect to a touch surface device and receive inputs from the touchsurface device. As discussed above, VR systems are wearable interactivesystems that allow a user to experience an artificial world. The usermay visually see a computer generated world through a display of the VRsystem.

In some instances, it may be more efficient to provide inputs using atouch screen device than using hand gestures in the artificial world ofthe VR system. However, when the user is wearing the head mounteddisplay (HMD) of the VR system, the user may not be able to see the realworld surroundings (which may include a touch screen device).

Examples herein provide a VR system that can be used to automaticallylocate a nearby touch surface device while the user is engaged in anartificial world of the VR system. Thus, the user may locate a touchscreen device, automatically connect to the touch screen device, and usethe touch screen device to provide inputs while in the artificial worldof the VR system. In other words, the user does not need to remove theHMD of the VR system to find, and connect to, the touch screen device.

FIG. 1 illustrates a block diagram of a VR system 100 of the presentdisclosure. In one example, the VR system 100 may include a head mounteddisplay (HMD) 102, a hand wearable interface 104, and a touch screendevice 106. The VR system 100 may include a processor 108 that may becommunicatively coupled to the hand wearable interface 104 via a wiredor wireless connection. In one example, the processor 108 may be locatedexternal to the HMD 102. In one example, as shown in FIG. 1, theprocessor 108 may be integrated as part of the HMD 102. The processor108 may execute various instructions stored in memory and/or functions,as described below.

As described above, sometimes using a touch screen device may be a moreaccurate way of providing inputs to the HMD 102 than using hand gesturesvia the hand wearable interface 104. However, when the HMD 102 is activeand displaying a computer generated artificial world on a display 112 ina virtual reality (VR) mode, the user may not be able to see where thetouch screen device 106 is located.

In one example, the touch screen device 106 may have a marker 120 thatcan be detected by the HMD 102 and/or the hand wearable interface 104.No other objects in the room may appear in the display 112 when the VRmode is active except the marker 120.

In one example, the marker 120 may be a pre-defined dynamic marker thatcan be detected by a camera 110 (e.g., a red, green, blue (RGB) videocamera, an infrared camera, and the like) coupled to the HMD 102. Forexample, a user may look around a room with the HMD 102 to see if thepre-defined dynamic marker appears in the display 112. The pre-defineddynamic marker may be a constantly changing mark or code, as opposed toa static or fixed code. The shape of the pre-defined dynamic marker maychange periodically, the colors of the pre-defined dynamic marker maychange periodically, the codes within the pre-defined dynamic marker maychange periodically, and the like.

In one example, the marker 120 may be an emission of a particularwavelength of light. For example, the camera 110 may be an infraredcamera and may detect an infrared wavelength of light emitted by thetouch screen device 106 that is invisible to the human eye so as not todistract non-users of the VR system 100. The infrared wavelength oflight may allow a non-user of the VR system 100 to use the touch screendevice 106 as a traditional device, while allowing the camera 110 tolocate the touch screen device 106. The infrared wavelength of light maybe shown on the display 112 such that the user may move towards thetouch screen device 106.

In one example, the touch screen device 106 may be registered with theprocessor 108 of the VR system 100. Thus, the VR system 100 may know thedimensions of the display of the touch screen device 106. The HMD 102 orthe hand wearable interface 104 may emit a signal to wake the touchscreen device 106. The marker 120 may be a glow of the display of thetouch screen device 106. The camera 110 may then identify the touchscreen device 106 by searching for a light emitted from a screen thathas the same dimensions as the touch screen device 106 that wasregistered with the VR system 100.

In one example, the hand wearable interface 104 may be used to locatethe touch screen device 106 without the marker 120. In one example, thehand wearable interface 104 may include an emitter 114. The emitter 114may be a wireless emitter that broadcasts a wireless signal.

When the touch screen device 106 receives the wireless signal, the touchscreen device 106 may transmit a response signal that is received by areceiver 116. The receiver 116 may be a wireless receiver that receivesthe response signal from the touch screen device 106. Examples of suchwireless links may include Bluetooth, radio frequency identification(RFID), near field communications (NFC), and the like. In one example,the distance to the touch screen device 106 may be calculated based onthe time to receive the response signal from the touch screen device106. When the response signal is received, an avatar of the touch screendevice 106 may be shown in the display 112. The avatar may grow smalleror larger as the user moves further away from or closer to the touchscreen device 106.

In one example, the hand wearable interface 104 may include a hapticfeedback 118. The haptic feedback 118 may be a component that providesfeedback (e.g., a vibration or a buzz) to the user when the handwearable interface 104 is located over or within a pre-defined distance(e.g., a few inches) from the touch screen device 106. In one example,the strength of the haptic feedback may vary and be proportional to thedetected proximity of the device. Thus, the user can locate and grab thetouch screen device 106 when the haptic feedback 118 provides constantfeedback.

In one example, after the user is done with the touch screen device 106,the user may place the touch screen device 106 at a location. The VRsystem 100 may remember the location where the touch screen device 106was placed such that the touch screen device 106 may be located morequickly the next time the user want to locate and use the touch screendevice 106.

Once the touch screen device 106 is located, the touch screen device 106may be held. The camera 110 may be used to detect an orientation of thetouch screen device 106. In one example, the touch screen device 106 mayhave sensors that can detect the orientation of the touch screen device106. The orientation of the touch screen device 106 can be transmittedto the processor 108 for display. After the touch screen device 106 islocated, held by the user, and the orientation is detected, the touchscreen device 106 may be communicatively coupled to the HMD 102. Thetouch screen device 106 may also be shown in the computer generatedartificial world in the orientation that is detected, as discussed infurther detail below. The touch screen device 106 can then be used toprovide inputs to the computer generated artificial world displayed onthe display 112.

FIG. 2 illustrates a block diagram of a VR system 200. The VR system 200may include similar components as the VR system 100 illustrated inFIG. 1. In one example, the HMD 102 may be communicatively coupled tothe processor 108. The HMD 102 may be to display the computer generatedartificial world when the HMD 102 is operating in a VR mode. In otherwords, a user may not be able to see his or her surroundings whenwearing the HMD 102 and interacting with the computer generatedartificial world in the VR mode.

The processor 108 may also be communicatively coupled to the handwearable interface 104. As discussed above, the processor 108 maycommunicate with the hand wearable interface 104 via a wired or wirelessconnection. The hand wearable interface 104 may be worn around the backof a user's hand to provide motion detection, gesture detection, and thelike to the processor 108.

The processor 108 may also be communicatively coupled to a wirelesscommunication interface 202. The wireless communication interface 202may establish a wireless connection 204 to the touch screen device 106.The processor 108 may then receive inputs from the touch screen device106 over the wireless connection 204.

In one example, the touch screen device 106 may be registered with a VRapplication executed by the processor 108, as noted above. In oneexample, a plurality of different touch screen devices 106 that areassociated with, or owned by, a user may be registered with the VRsystem 100. For example, the user may have a touch screen phone and atouch screen tablet device to use in the computer generated artificialworld.

In addition to learning the dimensions of the touch screen device 106,the registration process may be used to download an application on thetouch screen device 106 that works with the VR mode of the HMD 102. Forexample, the application may allow the touch screen device 106 toautomatically establish the wireless connection 204 with the processor108 when the touch screen device 106 is located.

The application may also allow the touch screen device 106 to trackwhich locations (e.g., x-y coordinates) of the display are touched andtransmit the location information to the processor 108. The processor108 may then identify a touch input displayed on an avatar of the touchscreen device 106 that is displayed in the computer generated artificialworld. The touch input may be associated with a function and thefunction may be executed in the computer generated artificial worldduring the VR mode of the HMD 102.

FIG. 3 illustrates an example display 112 of an avatar 306 of the touchscreen device 106 in the HMD 102 of the VR system 100. For example,after the touch screen device 106 is located and an indication that theuser is holding the touch screen device 106 is received, the display 112may show the avatar 306 of the touch screen device 106. The avatar 306may show the touch screen device 106 in an orientation in which the useris holding the touch screen device 106. The avatar 306 may also showtouch inputs 310, 312, and 314 of a graphical user interface (GUI) 308.

The dimensions of the avatar 306 may be similar to the dimensions of thetouch screen device 106. The size of the GUI 308 may be similar to thesize of a GUI that would be displayed on the touch screen device 106.The number and size of the touch inputs 310, 312, and 314 may be afunction of the size, or dimensions, of the touch screen device 106.

As shown in FIG. 3, the touch screen device 106 may be powered on andactive, but display a blank screen. In other words, the touch screendevice 106 may not show any information or GUI despite being activatedand connected to the HMD 102. In some examples, the touch screen device106 may be a touch screen device without a display. Since the touchscreen device 106 does not show any information, a touch screen devicewithout a display may be used with the VR system 100.

However, in the display 112 that shows the computer generated artificialworld, the avatar 306 may display the GUI 308 with touch inputs 310,312, and 314. It should be noted that although three touch inputs areillustrated in FIG. 3, any number of touch inputs may be displayed inthe avatar 306. In the computer generated artificial world, the user maywant to select the touch input 310. The user may touch a location on thetouch screen device 106 that is associated with the location of thetouch input 310 in the avatar 306. The touch screen device 106 maydetect the touch and record a location that is touched. The location maybe transmitted to the processor 108 of the HMD 102 via the wirelessconnection 204.

In one example, the touch screen device 106 may display imagesassociated with an application or operating system executed by the touchscreen device 106. The images on the touch screen device 106 may betransmitted to the processor 108 to be displayed on the display 112.Thus, the GUI 308 with the touch inputs 310, 312, and 314 may be what isshown on the touch screen device 106. In other words, the display 112may show what is actually displayed by the touch screen device 106. Thismay allow the user to check messages, notifications, and the like, onthe touch screen device 106 while using the touch screen device 106 inthe computer generated artificial world.

The processor 108 may determine which touch input was selected based onthe location information that is received from the touch screen device106. For example, the processor 108 may determine that the touch input310 was selected based on the location information from the touch screendevice 106.

The processor 108 may then determine a function that is associated withthe touch input 310. For example, the touch inputs 310, 312, and 314 maybe associated with different functions that can be executed in thecomputer generated world. For example, the computer generated world maybe a construction simulation. Each touch input 310, 312, and 314 may bea touch input to use a different tool, use a different vehicle, build adifferent structure, and the like. The touch input 310 may be associatedwith a function to demolish a structure. The processor 108 may determinethat the touch input 310 has been selected to demolish a structure. Theprocessor 108 may then prompt a user to select a structure in thecomputer generated artificial world and demolish the structure that isselected based on the selection of the touch input 310.

It should be noted that the inputs 310, 312, and 314 may be associatedwith other functions in different applications. For example, thecomputer generated artificial world may be a video game, a travelsimulator, and the like, and the inputs 310, 312, and 314 may executedifferent functions in the different computer generated artificialworlds.

As a result, the VR system 100 may allow a user to locate a touch screendevice 106 while using the HMD 102. Thus, the user does not need todeactivate a VR mode and remove the HMD 102 to locate the touch screendevice 106. In addition, the touch screen device 106 may be connected tothe HMD 102 and used to provide touch inputs in the computer generatedartificial world. For example, an avatar 306 of the touch screen device106 may be shown in the computer generated artificial world and used tointeract, or execute different functions, in the computer generatedartificial world.

FIG. 4 illustrates a flow diagram of an example method 400 forconnecting a virtual reality device to a touch surface device to receivean input from the touch surface device. In one example, the method 400may be performed by the virtual reality system 100, or the apparatus 500illustrated in FIG. 5 and described below.

At block 402, the method 400 begins. At block 404, the method 400receives an indication that a touch screen device that is located isbeing held. For example, the user may want to use the touch screendevice to provide inputs to a virtual reality world in a virtual realitysystem. The user may not be able to see his or her surroundings in thereal world while in the virtual reality world.

As a result, one of the methods described above can be used to locatethe touch screen device. In one example, the indication may be a hapticfeedback that is triggered when the touch screen device is locatedand/or held. In one example, the indication may be an audible indicationwhen the touch screen device is located and/or held. For example, a beepor tone may be played in the audio of the virtual reality system. In oneexample, the indication may be a visual indication when the touch screendevice is located and/or held. For example, an outline of the touchscreen device may flash or the touch screen device may be displayed inthe virtual reality system.

At block 406, the method 400 generates an avatar of the touch screendevice in a computer generated artificial world and causes the avatar tobe displayed in a head mounted display (HMD) of a virtual reality (VR)system that displays the computer generated artificial world. Forexample, a graphical representation of the touch screen device may begenerated and displayed in the HMD of the VR system.

In one example, the avatar of the touch screen device may display animage or interface that is not shown on the real touch screen device.For example, the touch screen device in reality may be powered on, butdisplay a blank screen. However, the avatar of the touch screen devicemay display a menu with different buttons that can be selected. The VRsystem may know the dimensions of the touch screen device via aregistration process. The registration process may also includedownloading an application on the touch screen device that works withthe VR system such that the touch screen device can identify andcommunicate selections of certain areas of the touch screen device tothe VR system.

Thus, the VR system may know how to size and locate the menu and buttonsin the avatar of the touch screen device. The location of a button inthe avatar of the touch screen device may correspond to a same locationon the real touch screen device.

At block 408, the method 400 establishes a wireless connection to thetouch screen device. The touch screen device may communicate wirelesslywith the virtual reality system to exchange data, inputs, and outputs.In one example, the wireless connection may be a Bluetooth low energy(BLE) connection, a Wi-Fi connection, a local area network (LAN)connection, and the like. The VR system may automatically initiate apairing process or process to establish the wireless connection when thetouch screen device is located and confirmed to be held.

At block 410, the method 400 receives a selection of a touch input onthe avatar of the touch screen device in the computer generatedartificial world. For example, a user may be playing a game in thecomputer generated artificial world. The touch screen device may be usedto make a selection in the game. A user may touch an area of the screenof the real touch screen device via interaction with the avatar of thetouch screen device in the computer generated artificial world.

The real touch screen device may detect a location of where the usertouched the display of the real touch screen device. The location may betransmitted to the VR system. The VR system may associate a touch inputon the avatar of the touch screen device with the location of the realtouch screen device that was touched.

At block 412, the method 400 executes a function associated with thetouch input in the computer generated artificial world. In one example,each button in the avatar of the touch screen device may be associatedwith a function within the computer generated artificial world. When theselection of the touch input is received in the block 410, the VR systemmay identify the function associated with the touch input that isselected. Using the game example above, selecting a touch input in thecomputer generated artificial world may equip a particular armament.Thus, when the armament touch input is selected in the computergenerated artificial world, the particular armament may be equipped inthe computer generated artificial world. At block 414, the method 400ends.

FIG. 5 illustrates an example of an apparatus 500. In one example, theapparatus 500 may be the virtual reality system 100. In one example, theapparatus 500 may include a processor 502 and a non-transitory computerreadable storage medium 504. The non-transitory computer readablestorage medium 504 may include instructions 506, 508, 510, 512, and 514that, when executed by the processor 502, cause the processor 502 toperform various functions.

In one example, the instructions 506 may include instructions to locatea touch screen device while a head mounted display (HMD) of a virtualreality (VR) system is displaying a computer generated artificial world.The instructions 508 may include instructions to establish a wirelessconnection to the touch screen device. The instructions 510 may includeinstructions to display an avatar of the touch screen device in thecomputer generated artificial world, wherein the avatar displays a touchinput that is different than what is displayed on the touch screendevice. The instructions 512 may include instructions to receive aselection of the touch input displayed on the avatar. The instructions514 may include instructions to execute a function associated with thetouch input in the computer generated artificial world.

It will be appreciated that variants of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be combined intomany other different systems or applications. Various presentlyunforeseen or unanticipated alternatives, modifications, variations, orimprovements therein may be subsequently made by those skilled in theart which are also intended to be encompassed by the following claims.

The invention claimed is:
 1. A virtual reality (VR) system, comprising:a head mounted display (HMD) to display a computer generated artificialworld; a hand wearable interface; a wireless communication interface toestablish a wireless communication path; and a processor communicativelycoupled to the HMD, the hand wearable interface, and the wirelesscommunication interface, the processor to receive an indication that atouch screen device is located from a locator device in the HMD or thehand wearable device, to establish a wireless connection to the touchscreen device via the wireless communication interface, and to receivean input via the touch screen device, wherein the touch screen device isregistered with the HMD such that the HMD can send a signal to the touchscreen device to wake the touch screen device to emit an indicator thatcan be detected by the HMD.
 2. The VR system of claim 1, wherein theindicator comprises a pre-defined dynamic marker and wherein the locatordevice comprises: a camera coupled to the HMD and communicativelycoupled to the processor to detect the pre-defined dynamic marker on thetouch screen device and transmit the pre-defined dynamic marker to theprocessor as the indication.
 3. The VR system of claim 1, wherein theindicator comprises a wavelength of light emitted by the touch screendevice and wherein the locator device comprises: a camera coupled to theHMD and communicatively coupled to the processor to detect thewavelength of light emitted by the touch screen device that is invisibleto a human eye and to transmit the wavelength of the light that isdetected to the processor as the indication.
 4. The VR system of claim1, wherein the locator device comprises: a wireless emitter located inthe hand wearable interface to emit a wireless signal; and a wirelessreceiver located in the hand wearable interface to receive a responsesignal from the touch screen device in response to the touch screendevice receiving the wireless signal.
 5. The VR system of claim 4,wherein the hand wearable interface comprises a haptic feedback deviceto provide haptic feedback when the hand wearable interface is locatedover the touch screen device.
 6. The VR system of claim 1, wherein theprocessor is to generate an avatar of the touch screen device when thetouch screen device is located and the HMD is to display the avatar inthe computer generated artificial world.
 7. The VR system of claim 6,wherein the processor is to generate touch screen input buttons for theavatar of the touch screen device and the HMD is to display the touchscreen input buttons on the avatar of the touch screen device in thecomputer generated artificial world.
 8. A non-transitory computerreadable storage medium encoded with instructions executable by aprocessor, the non-transitory computer-readable storage mediumcomprising: instructions to locate a touch screen device while a headmounted display (HMD) of a virtual reality (VR) system is displaying acomputer generated artificial world, wherein the touch screen device isregistered with the VR system such that the VR system can send a signalto the touch screen device to wake the touch screen device to emit anindicator that can be detected by the HMD; instructions to establish awireless connection to the touch screen device; instructions to displayan avatar of the touch screen device in the computer generatedartificial world, wherein the avatar displays a touch input that isdifferent than what is displayed on the touch screen device;instructions to receive a selection of the touch input displayed on theavatar; and instructions to execute a function associated with the touchinput in the computer generated artificial world.
 9. The non-transitorycomputer readable storage medium of claim 8, wherein the indicatorcomprises a pre-defined dynamic marker and wherein the instructions tolocate the touch screen device, comprises: instructions to detect thepre-defined dynamic marker on the touch screen device.
 10. Thenon-transitory computer readable storage medium of claim 8, wherein theindicator comprises a wavelength of light emitted by the touch screendevice and wherein the instructions to locate the touch screen device,comprises: instructions to detect the wavelength of light emitted by thetouch screen device that is invisible to a human eye.
 11. Thenon-transitory computer readable storage medium of claim 8, wherein theinstructions to locate the touch screen device, comprises: instructionsto emit a wireless signal that is received by the touch screen device;instructions to receive a response signal from the touch screen devicein response to the touch screen device receiving the wireless signal;instructions to calculate a distance to the touch screen device based onthe response signal.
 12. The non-transitory computer readable storagemedium of claim 11, further comprising: instructions to change a size ofthe avatar of the touch screen based on the distance as the distancechanges.
 13. A method, comprising: receiving, by a processor, anindication that a touch screen device that is located is being held,wherein the touch screen device is registered with a virtual reality(VR) system such that the VR system can send a signal to the touchscreen device to wake the touch screen device to emit an indicator thatcan be detected by a head mounted display (HMD) of the VR system;generating, by the processor, an avatar of the touch screen device in acomputer generated artificial world and causing the avatar to bedisplayed in the HMD that displays the computer generated artificialworld; establishing, by the processor, a wireless connection to thetouch screen device; receiving, by the processor, a selection of a touchinput on the avatar of the touch screen device in the computer generatedartificial world; and executing, by the processor, a function associatedwith the touch input in the computer generated artificial world.
 14. Themethod of claim 13, further comprising: registering, by the processor, aplurality of touch screen devices associated with a user by storing inmemory of the VR system an identification and a screen size of each oneof the plurality of touch screen devices.
 15. The method of claim 14,wherein the indicator comprises an area of light, the method furthercomprising: detecting, by the processor, the area of light that isapproximately equal to the screen size of the touch screen device tolocate the touch screen device.
 16. The VR system of claim 1, whereinthe processor is further to: record a location of the touch screendevice where the touch screen device is set down.